What is lost in translation from visual graphics to text for accessibility

Many blind and low-vision individuals are unable to access digital media visually. Currently, the solution to this accessibility problem is to produce text descriptions of visual graphics, which are then translated via text-to-speech screen reader technology. However, if a text description can accurately convey the meaning intended by an author of a visualization, then why did the author create the visualization in the first place? This essay critically examines this problem by comparing the so-called graphic–linguistic distinction to similar distinctions between the properties of sound and speech. It also presents a provisional model for identifying visual properties of graphics that are not conveyed via text-tospeech translations, with the goal of informing the design of more effective sonic translations of visual graphics

Research statement

My research lies in the domain of visual media, and graphic representation in particular. I seek a scientifically-grounded understanding of established and emerging conventions in the fine & applied visual arts. This is a perceptual-cognitive approach to understanding visual media

A cognitive exploration of the “non-visual” nature of geometric proofs

Why are Geometric Proofs (Usually) “Non-Visual”? We asked this question as a way to explore the similarities and differences between diagrams and text (visual thinking versus language thinking). Traditional text-based proofs are considered (by many to be) more rigorous than diagrams alone. In this paper we focus on human perceptual-cognitive characteristics that may encourage textual modes for proofs because of the ergonomic affordances of text relative to diagrams. We suggest that visual-spatial perception of physical objects, where an object is perceived with greater acuity through foveal vision rather than peripheral vision, is similar to attention navigating a conceptual visual-spatial structure. We suggest that attention has foveal-like and peripheral-like characteristics and that textual modes appeal to what we refer to here as foveal-focal attention, an extension of prior work in focused attention

Iconic Properties are Lost when Translating Visual Graphics to Text for Accessibility

For many blind and low-vision individuals, accessing charts and graphs often means accessing a text description of the graphics, usually aurally. However, in doing so, parts of the charts that are not originally conveyed textually are lost in the translation into text. By synthesizing ideas from the science and philosophy of perception and cognition, diagrammatic reasoning, and semiotics, this essay makes the case that translating charts into text descriptions results in the loss of iconic properties of the graphics, and proposes that non-linguistic sonification can be recruited to preserve such properties. The essay concludes by proposing how predictions based on this synthesis can inform design

The audio game laboratory: Building maps from games

Audio games demonstrate an emergence of interactive parameter mapping sonifications that potentially optimally display geographical information and a large number of simultaneous data variables. Our preliminary investigation of audio games is in response to a call for more research on parameter mapping sonifications, such as the best way of presenting auditory legends for representations, effectiveness of spatial audio, map comprehension techniques, and finding optimal sonic variable mappings. We also present a proposed set of auditory map interfaces observed in audio games. Commercially available interactive interfaces and audio games – that have been shaped and informally “tested” by the selection pressures of a demanding consumer market – can serve as examples of potentially effective conventions informing future work in the auditory display research community

Mobility in Europe: Analysis of the 2005 Eurobarometer Survey on Geographical and Labour Market Mobility

The European Commission has designated the year 2006 as 'European Year of Workers' Mobility'. The purpose of the initiative is to inform EU citizens of the benefits and the costs of both geographical mobility and job or labour market mobility; the realities of working in another country or changing job or career; and the rights they are entitled to as migrant workers. The initiative also aims to promote the exchange of good practice between public authorities and institutions, the social partners and the private sector, and to promote greater study of the scale and nature of geographical and job mobility within the Union

Progress toward sonifying Napoleon’s march and fluid flow simulations through binaural horizons

Cross-modal data analytics—that can be rendered for experience through vision, hearing, and touch—poses a fundamental challenge to designers. Non-linguistic sonification is a well-researched means for non-visual pattern recognition but higher density datasets pose a challenge. Because human hearing is optimized for detecting locations on a horizontal plane, our approach recruits this optimization by employing an immersive binaural horizontal plane using auditory icons. Two case studies demonstrate our approach: A sonic translation of a map and a sonic translation of a computational fluid dynamics simulation